Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

implement data_type_code_to_name on SparkConnectionManager #647

Merged
merged 9 commits into from
Mar 3, 2023
Merged

implement data_type_code_to_name on SparkConnectionManager #647

merged 9 commits into from
Mar 3, 2023

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Feb 22, 2023 •
edited
Loading

resolves #639

Description

Checklist

🎩 differing sql and schema.yml schema
SQL:

---test_schema.sql
SELECT 1 as a, '123' as b, array(1,2,3) as int_array, current_date() as date

schema with a bunch of wrong data_type values:

models:
  - name: test_schema
    config:
      contract: true
    columns:
      - name: a
        data_type: STRING
      - name: b
        data_type: INT
      - name: int_array
        data_type: INT
      - name: date
        data_type: STRING

❯ dbt run --select test_schema --project-dir ~/src/jaffle_shop;
23:33:10  Running with dbt=1.5.0-b1
23:33:10  Found 7 models, 18 tests, 0 snapshots, 1 analysis, 549 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 1 metric, 3 groups
23:33:10  
23:33:13  Concurrency: 1 threads (target='dev')
23:33:13  
23:33:13  1 of 1 START sql table model analytics.test_schematest_v2 ...................... [RUN]
23:33:15  1 of 1 ERROR creating sql table model analytics.test_schematest_v2 ............. [ERROR in 1.51s]
23:33:15  
23:33:15  Finished running 1 table model in 0 hours 0 minutes and 5.31 seconds (5.31s).
23:33:15  
23:33:15  Completed with 1 error and 0 warnings:
23:33:15  
23:33:15  Compilation Error in model test_schema (models/test_schema.sql)
23:33:15    Please ensure the name, data_type, order, and number of columns in your `yml` file match the columns in your SQL file.
23:33:15    Schema File Columns: a STRING_TYPE, b INT_TYPE, int_array INT_TYPE, date STRING_TYPE
23:33:15    
23:33:15    SQL File Columns: a INT_TYPE, b STRING_TYPE, int_array ARRAY_TYPE, date DATE_TYPE

🎩 fixing the incorrect yaml schema:

models: 
  - name: test_schema
    config:
      contract: true
    columns:
      - name: a
        data_type: INT
      - name: b
        data_type: STRING
      - name: int_array
        data_type: ARRAY<INT>
      - name: date
        data_type: DATE

❯ dbt run --select test_schema --project-dir ~/src/jaffle_shop;
23:37:15  Running with dbt=1.5.0-b1
23:37:16  Found 7 models, 18 tests, 0 snapshots, 1 analysis, 549 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 1 metric, 3 groups
23:37:16  
23:37:17  Concurrency: 1 threads (target='dev')
23:37:17  
23:37:17  1 of 1 START sql table model analytics.test_schematest_v2 ...................... [RUN]
23:37:21  1 of 1 OK created sql table model analytics.test_schematest_v2 ................. [OK in 3.95s]
23:37:21  
23:37:21  Finished running 1 table model in 0 hours 0 minutes and 5.85 seconds (5.85s).
23:37:21  
23:37:21  Completed successfully
23:37:21  
23:37:21  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

@cla-bot cla-bot bot added the cla:yes label Feb 22, 2023
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-spark contributing guide.

@MichelleArk MichelleArk marked this pull request as ready for review February 28, 2023 23:39
@MichelleArk MichelleArk requested a review from a team as a code owner February 28, 2023 23:39
@MichelleArk MichelleArk merged commit a405d24 into main Mar 3, 2023
@MichelleArk MichelleArk deleted the CT-2104/get_column_schema_from_query branch March 3, 2023 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mikealfare mikealfare mikealfare approved these changes

@VersusFacit VersusFacit Awaiting requested review from VersusFacit

Assignees
No one assigned
Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CT-2104] Implement get_column_schema_from_query macro
3 participants
@MichelleArk @mikealfare @colin-rogers-dbt